Method and system for identifying biometric characteristics using machine learning techniques

ABSTRACT

A method and system may use machine learning analysis of audio data to automatically identify a user&#39;s biometric characteristics. A user&#39;s client computing device may capture audio of the user. Feature data may be extracted from the audio and applied to statistical models for determining several biometric characteristics. The determined biometric characteristic values may be used to identify individual health scores and the individual health scores may be combined to generate an overall health score and longevity metric. An indication of the user&#39;s biometric characteristics which may include the overall health score and longevity metric may be displayed on the user&#39;s client computing device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 16/667,322, filed on Oct. 29, 2019, entitled“Method and System for Identifying Biometric Characteristics UsingMachine Learning Techniques,” which claims priority to U.S. applicationSer. No. 15/837,522 filed on Dec. 11, 2017, entitled “Method and Systemfor Identifying Biometric Characteristics Using Machine LearningTechniques,” now known as U.S. Pat. No. 10,503,970, issued on Dec. 10,2019, the entire contents of which is hereby expressly incorporatedherein by reference.

TECHNICAL FIELD

The present disclosure generally relates to identifying biometriccharacteristics and, more particularly to utilizing computer visiontechniques and machine learning techniques to predict a user's biometriccharacteristics based on a video of the user.

BACKGROUND

Today, a user's health status may be determined based on severalbiometric characteristics, such as the user's age, gender, bloodpressure, heart rate, body mass index (BMI), body temperature, stresslevels, smoking status, etc. These biometric characteristics aretypically obtained through self-reporting from the user (e.g., byfilling out a form indicating the user's gender, birth date, etc.)and/or medical examinations that include taking measurements conductedby various instruments, such as a thermometer, scale, heart ratemonitor, blood pressure cuff, etc.

This process of filling out forms and taking measurements with severaldifferent instruments may be difficult and time consuming for the user.Users may also withhold information or report incorrect informationwhich may lead to inaccuracies in the health status assessment (e.g.,from errors in self-reporting or uncalibrated instruments).

SUMMARY

To efficiently and accurately predict a user's health status andcorresponding longevity metric, a biometric characteristic system may betrained using various machine learning techniques to create predictivemodels for determining biometric characteristics of the user based onvideo of the user. The determined or predicted biometric characteristicsmay be combined to generate an overall indication of the user's healthwhich may be used to generate a longevity metric for the user. Thebiometric characteristic system may be trained by obtaining audiovisualdata (e.g., videos or images) of several people having known biometriccharacteristics at the time the audiovisual data is captured (e.g., age,gender, BMI, etc.). The people may be referred to herein as “trainingsubjects.” For example, the training data may include public audiovisualdata such as movies, television, music videos, etc., featuring famousactors or actresses having biometric characteristics which are known orwhich are easily obtainable through public content (e.g., via InternetMovie Database (IMDb®), Wikipedia™, etc.).

In some embodiments, the training data may include feature dataextracted from the audiovisual data using computer vision techniques andthe training data may include the known biometric characteristics thatcorrespond to each set of feature data. In any event, the training datamay be analyzed using various machine learning techniques to generatepredictive models which may be used to determine biometriccharacteristics of a user, where the user's biometric characteristicsare unknown to the system.

After the training period, a user may capture audiovisual data such as avideo of herself via a client computing device and provide the video tothe biometric characteristic system. The biometric characteristic systemmay analyze the video using computer vision techniques to identify aportion of each frame that corresponds to the user's face and to extractfeature data from the identified portions. The extracted feature datafor the user may be compared to the predictive models to determine theuser's biometric characteristics. Additionally, the biometriccharacteristics may be used to determine an overall health indicator forthe user and/or a longevity metric. Then the biometric characteristics,the overall health indicator, and/or the longevity metric may beprovided for display on the user's client computing device.

In this manner, a user's health status may be predicted efficiently(e.g., in real-time or at least near-real time from when the video isprovided to the biometric characteristic system) and accurately withoutrelying on self-reporting, medical examinations, or readings fromvarious instruments. The present embodiments advantageously streamlinethe health status assessment process and increase ease of use for userswho may simply submit a short video clip of themselves instead ofengaging in a lengthy process of filling out forms and providing medicalrecords. Moreover, by capturing video rather than still images, thepresent embodiments advantageously extract movement data which may beused to predict additional biometric characteristics such as heart rate,blood pressure, galvanic skin response (GSR), etc. Furthermore, videomay be more difficult for users to modify in attempts to alter theirphysical appearances, and therefore using video may prevent fraud.

In an embodiment, a method for identifying biometric characteristics ofa user based on audio data is provided. The method includes obtaining aplurality of sets of audio data corresponding to a plurality of peopleand one or more biometric characteristics for each of the plurality ofpeople. For each of the one or more biometric characteristics, themethod includes analyzing the plurality of sets of audio data toidentify a plurality of features and generate a model for determining anunknown biometric characteristic of a user based on the plurality offeatures and the obtained biometric characteristic for each of theplurality of people. The method also includes receiving a set of audiodata corresponding to a user, wherein the audio data includes voice datacaptured over a threshold time period, applying features within the setof audio data corresponding to the user to the one or more models todetermine the one or more biometric characteristics of the user, andproviding an indication of the determined one or more biometriccharacteristics of the user to a client computing device.

In another embodiment, a server computing device for identifyingbiometric characteristics of a user based on audio data is provided. Theserver computing device includes one or more processors and anon-transitory computer-readable memory coupled to the one or moreprocessors and storing instructions thereon. When executed by the one ormore processors, the instructions cause the server computing device toobtain a plurality of sets of audio data corresponding to a plurality ofpeople and one or more biometric characteristics for each of theplurality of people. For each of the one or more biometriccharacteristics, the instructions cause the server computing device toanalyze the plurality of sets of audio data to identify a plurality offeatures and generate a model for determining an unknown biometriccharacteristic of a user based on the plurality of features and theobtained biometric characteristic for each of the plurality of people.The instructions further cause the server computing device to receive aset of audio data corresponding to a user, wherein the audio dataincludes voice data captured over a threshold time period, applyfeatures within the set of audio data corresponding to the user to theone or more models to determine the one or more biometriccharacteristics of the user, and provide an indication of the determinedone or more biometric characteristics of the user to a client computingdevice.

In yet another embodiment, a non-transitory computer-readable memory isprovided. The computer-readable memory stores instructions thereon. Whenexecuted by one or more processors, the instructions cause the one ormore processors to obtain a plurality of sets of audio datacorresponding to a plurality of people and one or more biometriccharacteristics for each of the plurality of people. For each of the oneor more biometric characteristics, the instructions cause the one ormore processors to analyze the plurality of sets of audio data toidentify a plurality of features and generate a model for determining anunknown biometric characteristic of a user based on the plurality offeatures and the obtained biometric characteristic for each of theplurality of people. The instructions further cause the one or moreprocessors to receive a set of audio data corresponding to a user,wherein the audio data includes voice data captured over a thresholdtime period, apply features within the set of audio data correspondingto the user to the one or more models to determine the one or morebiometric characteristics of the user, and provide an indication of thedetermined one or more biometric characteristics of the user to a clientcomputing device.

BRIEF DESCRIPTION OF THE DRAWINGS

The figures described below depict various aspects of the system andmethods disclosed therein. It should be understood that each figuredepicts an embodiment of a particular aspect of the disclosed system andmethods, and that each of the figures is intended to accord with apossible embodiment thereof. Further, wherever possible, the followingdescription refers to the reference numerals included in the followingfigures, in which features depicted in multiple figures are designatedwith consistent reference numerals.

FIG. 1 illustrates a block diagram of a computer network and system onwhich an exemplary biometric characteristic system may operate inaccordance with the presently described embodiments;

FIG. 2A illustrates a block diagram of an exemplary biometriccharacteristic server that can operate in the system of FIG. 1 ;

FIG. 2B illustrates a block diagram of an exemplary client computingdevice that can operate in the system of FIG. 1 ;

FIGS. 3A-B illustrate exemplary training data including video framesdepicting faces which may be used for training a training module;

FIG. 4 illustrates an exemplary image analysis of feature data overseveral video frames in accordance with the presently describedembodiments;

FIG. 5 illustrates an exemplary video capturing screen of a clientapplication in accordance with the presently described embodiments;

FIG. 6 illustrates a flow diagram representing an exemplary method foridentifying biometric characteristics of a user based on audiovisualdata in accordance with the presently described embodiments; and

FIG. 7 illustrates a flow diagram representing an exemplary method forcapturing audiovisual data representing a user and presentingindications of automatically determined biometric characteristics of theuser in accordance with the presently described embodiments.

DETAILED DESCRIPTION

Although the following text sets forth a detailed description ofnumerous different embodiments, it should be understood that the legalscope of the description is defined by the words of the claims set forthat the end of this patent and equivalents. The detailed description isto be construed as exemplary only and does not describe every possibleembodiment since describing every possible embodiment would beimpractical. Numerous alternative embodiments could be implemented,using either current technology or technology developed after the filingdate of this patent, which would still fall within the scope of theclaims.

It should also be understood that, unless a term is expressly defined inthis patent using the sentence “As used herein, the term ‘______’ ishereby defined to mean . . . ” or a similar sentence, there is no intentto limit the meaning of that term, either expressly or by implication,beyond its plain or ordinary meaning, and such term should not beinterpreted to be limited in scope based on any statement made in anysection of this patent (other than the language of the claims). To theextent that any term recited in the claims at the end of this patent isreferred to in this patent in a manner consistent with a single meaning,that is done for sake of clarity only so as to not confuse the reader,and it is not intended that such claim term be limited, by implicationor otherwise, to that single meaning. Finally, the patent claims at theend of this patent application are not intended to be construed under 35U.S.C. § 112(f) unless traditional means-plus-function language isexpressly recited, such as “means for” or “step for” language beingexplicitly recited in the claim(s). The systems and methods describedherein are directed to an improvement to computer functionality, andimprove the functioning of conventional computers.

Accordingly, as used herein, the term “training subject” may refer to aperson depicted in a video or other set of audiovisual data, where theperson has biometric characteristics that are known to the system. Forexample, the training subject may be an actor or actress whose height,weight, age, gender, etc., may be retrieved from IMDb®, Wikipedia™, orany other suitable source of public content. Portions of each videoframe depicting the training subject's face may be analyzed along withportions of other video frames depicting the faces of other trainingsubjects to generate a statistical model for predicting a biometriccharacteristic based on the videos.

The term “feature” or “feature data” as used herein may be used to referto an image feature extracted from a video frame or other image includedin the audiovisual data. An image feature may include a line, edge,shape, object, etc. The feature may be described by a feature vectorthat includes attributes of the feature, such as RGB pixel values forthe feature, the position of the feature within the face frame, the sizeof the feature relative to the face frame, the shape of the feature, thetype of feature, pixel distances between the feature and other features,or any other suitable attributes.

The term “biometric characteristic” as used herein may refer to abiographical or physiological trait of a person, such as age, gender,BMI, blood pressure, heart rate, GSR, smoking status, body temperature,etc. Each biometric characteristic may correspond to a range ofbiometric characteristic values. For example, the biometriccharacteristic “age” may have biometric characteristic values from 1 to120.

The term “longevity metric” as used herein may be used to refer to anestimate of the user's life expectancy or a remaining life expectancyfor the user. The longevity metric may also be a monthly or yearly lifeinsurance premium quote based on the remaining life expectancy for theuser and/or other factors, such as the coverage amount, the policy type(e.g., term life insurance or whole life insurance), etc.

Generally speaking, techniques for determining biometric characteristicsmay be implemented in one or several client computing devices, one orseveral network servers or a system that includes a combination of thesedevices. However, for clarity, the examples below focus primarily on anembodiment in which a biometric characteristic server obtains a set oftraining data and uses the training data to generate statistical modelsfor determining biometric characteristics of a user to generate alongevity metric for the user. The statistical models may be generatedbased on audiovisual data representing faces of training subjects havingbiometric characteristics known to the system and based on the knownbiometric characteristic values for each training subject. In someembodiments, the statistical models may be generated based on featuredata included within the audiovisual data. Various machine learningtechniques may be used to train the biometric characteristic server.

After the biometric characteristic server has been trained, a user maycapture a video of herself taken over a threshold time period (e.g.,five seconds, ten seconds, a minute, etc.) on the user's clientcomputing device. The client computing device may transmit the video tothe biometric characteristic server which may analyze the video framesto identify the user's face within each video frame. The biometriccharacteristic server may then identify feature data and may analyze thefeature data using the machine learning techniques to determinebiometric characteristics of the user. In some embodiments, thebiometric characteristics server may use the biometric characteristicsto determine a longevity metric for the user. An indication of thebiometric characteristics and/or an indication of the longevity metricmay be transmitted for display on the client computing device.

Referring to FIG. 1 , an example biometric characteristic system 100includes a biometric characteristic server 102 and a plurality of clientcomputing devices 106-116 which may be communicatively connected througha network 130, as described below. According to embodiments, thebiometric characteristic server 102 may be a combination of hardware andsoftware components, also as described in more detail below. Thebiometric characteristic server 102 may have an associated database 124for storing data related to the operation of the biometriccharacteristic system 100 (e.g., training data including audiovisualdata such as video representing training subject's faces, feature dataextracted from video frames, actual biometric characteristics for thetraining subjects, etc.). Moreover, the biometric server 102 may includeone or more processor(s) 132 such as a microprocessor coupled to amemory 140.

The memory 140 may be tangible, non-transitory memory and may includeany types of suitable memory modules, including random access memory(RAM), read-only memory (ROM), flash memory, other types of persistentmemory, etc. The memory 140 may store, for example instructionsexecutable on the processors 132 for a training module 134 and abiometric identification module 136. The biometric characteristic server102 is described in more detail below with reference to FIG. 2A.

To generate statistical models for determining biometriccharacteristics, a training module 134 may obtain a set of training databy receiving videos or other audiovisual data of several trainingsubjects where each video is captured over a threshold period of time.The video or other audiovisual data may be used to extract feature datafrom portions of the video frames that depict a training subject's face.The training module 134 may also obtain biometric characteristic valuesfor the training subject. For example, the training subject may be a 35year-old male having a BMI of 31 and blood pressure of 130/90. Thetraining module 134 may then analyze the audiovisual data and knownbiometric characteristic values to generate a statistical model for aparticular biometric characteristic (e.g., age). In some embodiments,the training module 134 may generate a statistical model for each ofseveral biometric characteristics (e.g., age, gender, BMI, bloodpressure, heart rate, GSR, smoking status, body temperature, etc.).

In any event, the set of training data may be analyzed using variousmachine learning techniques, such as neural networks, deep learning,naïve Bayes, support vector machines, linear regression, polynomialregression, logistic regression, random forests, boosting, nearestneighbors, etc. In some embodiments, the statistical models may begenerated using different machine learning techniques. For example, thestatistical model for predicting age may be generated using deeplearning and the statistical model for predicting gender may begenerated using naïve Bayes. In other embodiments, each statisticalmodel may be generated using the same machine learning technique (e.g.,deep learning). In a testing phase, the training module 134 may comparetest audiovisual data for a test user to the statistical models todetermine biometric characteristics of the test user.

If the training module 134 makes the correct determination morefrequently than a predetermined threshold amount, the statistical modelmay be provided to a biometric identification module 136. On the otherhand, if the training module 134 does not make the correct determinationmore frequently than the predetermined threshold amount, the trainingmodule 134 may continue to obtain training data for further training.

The biometric identification module 136 may obtain the statisticalmodels for each biometric characteristic as well as audiovisual data fora user captured over a threshold period of time, such as a five-secondvideo of the user. For example, the biometric identification module 136may receive the audiovisual data from one of the client computingdevices 106-116. The audiovisual data for the user may be compared tothe statistical models to determine biometric characteristic values forthe user. In some embodiments, the biometric identification module 136may determine a likelihood that a biometric characteristic of the useris a particular value. For example, the biometric identification module136 may determine there is a 70 percent chance the user is male and a 20percent chance the user is female.

The biometric identification module 136 may then utilize the determinedbiometric characteristic values for the user or the likelihoods ofbiometric characteristic values to determine an overall health indicatorfor the user and/or a longevity metric for the user. For example, eachbiometric characteristic value may be associated with an individualhealth score. The individual health scores may be combined and/oraggregated in any suitable manner to determine an overall health scoreas the overall health indicator. In some embodiments, an individualhealth score may be determined using a lookup table based on thebiometric characteristic value. The rules for determining the overallhealth score from the individual health scores may also be included inthe lookup table. In other embodiments, the overall health score may bedetermined using machine learning techniques by generating a statisticalmodel for determining the overall health score based on individualbiometric characteristics or health scores. This is described in moredetail below.

In any event, the overall health indicator may correspond to aparticular longevity metric, where higher overall health indicatorscorrespond to higher amounts of longevity. In some embodiments, thelongevity metric may be an estimate of the user's life expectancy or aremaining life expectancy for the user. In other embodiments, thelongevity metric may be a monthly or yearly life insurance premium quotebased on the estimated amount of longevity for the user, the coverageamount, the policy type (e.g., term life insurance or whole lifeinsurance), etc.

The biometric identification module 136 may transmit an indication ofthe biometric characteristics to one of the client computing devices106-116 for display on a user interface. The indication may include thebiometric characteristic values, the individual health scores, theoverall health score, the longevity metric including a monthly or yearlylife insurance premium quote, or any other suitable indication of thebiometric characteristics of the user.

The client computing devices 106-116 may include, by way of example,various types of “mobile devices,” such as a tablet computer 106, a cellphone 108, a personal digital assistant (PDA) 110, a smart phone 112, alaptop computer 114, a desktop computer 116, a portable media player(not shown), a home phone, a pager, a wearable computing device, smartglasses, smart watches or bracelets, phablets, other smart devices,devices configured for wired or wireless RF (Radio Frequency)communication, etc. Of course, any client computing device appropriatelyconfigured may interact with the biometric characteristic system 100.The client computing devices 106-116 need not necessarily communicatewith the network 130 via a wired connection. In some instances, theclient computing devices 106-116 may communicate with the network 130via wireless signals 120 and, in some instances, may communicate withthe network 130 via an intervening wireless or wired device 118, whichmay be a wireless router, a wireless repeater, a base transceiverstation of a mobile telephony provider, etc.

Each of the client computing devices 106-116 may interact with thebiometric characteristic server 102 to receive web pages and/or serverdata and may display the web pages and/or server data via a clientapplication and/or an Internet browser (described below). For example,the smart phone 112 may display a video capturing screen 122, maycapture video of a user, and may interact with the biometriccharacteristic server 102. For example, when a user captures video ofherself, the video may be transmitted to the biometric characteristicserver 102.

The biometric characteristic server 102 may communicate with the clientcomputing devices 106-116 via the network 130. The digital network 130may be a proprietary network, a secure public Internet, a local areanetwork (LAN), a wide area network (WAN), a virtual private network(VPN) or some other type of network, such as dedicated access lines,plain ordinary telephone lines, satellite links, combinations of these,etc. Where the digital network 130 comprises the Internet, datacommunication may take place over the digital network 130 via anInternet communication protocol.

Turning now to FIG. 2A, the biometric characteristic server 102 mayinclude a controller 224. The controller 224 may include a programmemory 226, a microcontroller or a microprocessor (MP) 228, arandom-access memory (RAM) 230, and/or an input/output (I/O) circuit234, all of which may be interconnected via an address/data bus 232. Insome embodiments, the controller 224 may also include, or otherwise becommunicatively connected to, a database 239 or other data storagemechanism (e.g., one or more hard disk drives, optical storage drives,solid state storage devices, etc.). The database 239 may include datasuch as training data, web page templates and/or web pages, and otherdata necessary to interact with users through the network 130. It shouldbe appreciated that although FIG. 2A depicts only one microprocessor228, the controller 224 may include multiple microprocessors 228.Similarly, the memory of the controller 224 may include multiple RAMs230 and/or multiple program memories 226. Although FIG. 2A depicts theI/O circuit 234 as a single block, the I/O circuit 234 may include anumber of different types of I/O circuits. The controller 224 mayimplement the RAM(s) 230 and/or the program memories 226 assemiconductor memories, magnetically readable memories, and/or opticallyreadable memories, for example.

As shown in FIG. 2A, the program memory 226 and/or the RAM 230 may storevarious applications for execution by the microprocessor 228. Forexample, a user-interface application 236 may provide a user interfaceto the biometric characteristic server 102, which user interface may,for example, allow a system administrator to configure, troubleshoot, ortest various aspects of the server's operation. A server application 238may operate to receive audiovisual data for a user, determine biometriccharacteristics of the user, and transmit an indication of the biometriccharacteristics to a user's client computing device 106-116. The serverapplication 238 may be a single module 238 or a plurality of modules238A, 238B such as the training module 134 and the biometricidentification module 136.

While the server application 238 is depicted in FIG. 2A as including twomodules, 238A and 238B, the server application 238 may include anynumber of modules accomplishing tasks related to implementation of thebiometric characteristic server 102. Moreover, it will be appreciatedthat although only one biometric characteristic server 102 is depictedin FIG. 2A, multiple biometric characteristic servers 102 may beprovided for the purpose of distributing server load, serving differentweb pages, etc. These multiple biometric characteristic servers 102 mayinclude a web server, an entity-specific server (e.g. an Apple® server,etc.), a server that is disposed in a retail or proprietary network,etc.

Referring now to FIG. 2B, the smart phone 112 (or any of the clientcomputing devices 106-116) may include a display 240, a communicationunit 258, accelerometers (not shown), a positioning sensor such as aGlobal Positioning System (GPS) (not shown), a user-input device (notshown), and, like the biometric characteristic server 102, a controller242. The client computing device 112 may also include an image sensor280 which may be a standard camera or a high resolution camera (e.g.,having a resolution of greater than 30 Megapixels). In some embodiments,the image sensor 280 may be removably attached to the exterior of theclient computing device 112. In other embodiments, the image sensor 280may be contained within the client computing device 112. Also in someembodiments, the image sensor 280 may capture images and video and maybe communicatively coupled to an audio sensor (not shown) such as amicrophone and speakers for capturing audio input and providing audiooutput.

Similar to the controller 224, the controller 242 may include a programmemory 246, a microcontroller or a microprocessor (MP) 248, arandom-access memory (RAM) 250, and/or an input/output (I/O) circuit254, all of which may be interconnected via an address/data bus 252. Theprogram memory 246 may include an operating system 260, a data storage262, a plurality of software applications 264, and/or a plurality ofsoftware routines 268. The operating system 260, for example, mayinclude one of a plurality of mobile platforms such as the iOS®,Android™, Palm® webOS, Windows Mobile/Phone, BlackBerry® OS, or Symbian®OS mobile technology platforms, developed by Apple Inc., Google Inc.,Palm Inc. (now Hewlett-Packard Company), Microsoft Corporation, Researchin Motion (RIM), and Nokia, respectively.

The data storage 262 may include data such as user profiles, applicationdata for the plurality of applications 264, routine data for theplurality of routines 268, and/or other data necessary to interact withthe biometric characteristic server 102 through the digital network 130.In some embodiments, the controller 242 may also include, or otherwisebe communicatively connected to, other data storage mechanisms (e.g.,one or more hard disk drives, optical storage drives, solid statestorage devices, etc.) that reside within the client computing device112.

The communication unit 258 may communicate with the biometriccharacteristic server 102 via any suitable wireless communicationprotocol network, such as a wireless telephony network (e.g., GSM, CDMA,LTE, etc.), a Wi-Fi network (802.11 standards), a WiMAX network, aBluetooth network, etc. The user-input device (not shown) may include a“soft” keyboard that is displayed on the display 240 of the clientcomputing device 112, an external hardware keyboard communicating via awired or a wireless connection (e.g., a Bluetooth keyboard), an externalmouse, or any other suitable user-input device.

As discussed with reference to the controller 224, it should beappreciated that although FIG. 2B depicts only one microprocessor 248,the controller 242 may include multiple microprocessors 248. Similarly,the memory of the controller 242 may include multiple RAMs 250 and/ormultiple program memories 246. Although the FIG. 2B depicts the I/Ocircuit 254 as a single block, the I/O circuit 254 may include a numberof different types of I/O circuits. The controller 242 may implement theRAM(s) 250 and/or the program memories 246 as semiconductor memories,magnetically readable memories, and/or optically readable memories, forexample.

The one or more processors 248 may be adapted and configured to executeany one or more of the plurality of software applications 264 and/or anyone or more of the plurality of software routines 268 residing in theprogram memory 242, in addition to other software applications. One ofthe plurality of applications 264 may be a client application 266 thatmay be implemented as a series of machine-readable instructions forperforming the various tasks associated with receiving information at,displaying information on, and/or transmitting information from theclient computing device 112.

One of the plurality of applications 264 may be a native applicationand/or web browser 270, such as Apple's Safari®, Google Chrome™,Microsoft Internet Explorer®, and Mozilla Firefox® that may beimplemented as a series of machine-readable instructions for receiving,interpreting, and/or displaying web page information from the server 102while also receiving inputs from the user. Another application of theplurality of applications may include an embedded web browser 276 thatmay be implemented as a series of machine-readable instructions forreceiving, interpreting, and/or displaying web page information from thebiometric characteristic server 102. One of the plurality of routinesmay include a video capturing routine 272 which captures several videoframes over a threshold time period (e.g., five seconds, ten seconds, aminute, etc.). Another routine in the plurality of routines may includea biometric characteristic display routine 274 which transmits the videoto the biometric characteristic server 102 and presents an indication ofthe user's biometric characteristics based on the video of the user.

Preferably, a user may launch the client application 266 from the clientcomputing device 112, to communicate with the biometric characteristicserver 102 to implement the biometric characteristic system.Additionally, the user may also launch or instantiate any other suitableuser interface application (e.g., the native application or web browser270, or any other one of the plurality of software applications 264) toaccess the biometric characteristic server 102 to realize the biometriccharacteristic system.

FIG. 3A depicts exemplary video frames 300 of training subjects havingknown biometric characteristics. Each of the video frames 300 may befrom public audiovisual data such as movies, television, music videos,etc., featuring famous actors or actresses having biometriccharacteristics which are known or which are easily obtainable throughpublic content (e.g., via IMDb®, Wikipedia™, etc.). For example, thefirst video frame 302 may depict John Doe who is a 40 year-old male andis 5′11″ tall and weighs 170 pounds. Based on his height and weight JohnDoe's BMI is 23.7. The video frames 300 and known biometriccharacteristics may be stored in a database 124 and used as trainingdata for the training module 134 as shown in FIG. 1 to generatestatistical models for determining biometric characteristics. While theexample video frames 300 include a single frame of each of severaltraining subjects, the video frames 300 may include several frames ofeach training subject to detect movement data and/or identify additionalfeatures for each training subject. Several video frames of the sametraining subject may be stored in the database 124 in association witheach other so that the biometric characteristic system 100 may analyzethe several video frames together to identify movement, detect theboundaries of the training subject's face, or for any other suitablepurpose.

In some embodiments, each video frame 300 may be analyzed using facedetection techniques to identify a portion of each video frame thatdepicts a training subject's face. Face detection techniques may includeedge detection, pixel entropy, blink detection, motion detection, skincolor detection, any combination of these, or any other computer visiontechniques. FIG. 3B depicts example video frames 320 similar to theexample video frames 300 annotated with outlines of the trainingsubjects' faces. For example, the first video frame 302 depicting JohnDoe includes an elliptical annotation 322 around the boundaries ofJohn's face. In some embodiments, the training module 134 may filter outthe remaining portion of the video frame that does not include JohnDoe's face and may store the annotated portion of the video frame 322 inthe database 124 for further analysis to generate the statistical modelsfor determining biometric characteristics. The training module 134 mayfilter each of the video frames 320 and may store the annotated portionsthat depict the training subjects' faces.

Then each of the portions of the video frames depicting the trainingsubjects' faces (referred to herein as “face frames”) may be furtheranalyzed to identify feature data within the face frames. Movement datamay also be identified indicating a change in the positions of thefeature data over multiple face frames for the same training subject.The feature data and the movement data for a training subject may thenbe stored in association with the biometric characteristics of thecorresponding training subject. To generate a statistical model for aparticular biometric characteristic (e.g., age), the training module 134may obtain the feature data and movement data associated with differentages to identify particular sets of feature data and/or movement datathat may be used to distinguish between ages (e.g., 45 or 55) or ageranges (e.g., 10-20 or 30-40). For example, while bones typically stopgrowing after puberty, cartilage such as ears and noses continue to growthroughout a person's life. Therefore, older people on average may havea larger ratio of the size of their ears and noses to the size of theirheads than younger people. This ratio may be included in the featuredata and used to distinguish between ages or age ranges. However, thisis merely one example for ease of understanding. Additional oralternative feature data and movement data may be used to generate thestatistical model for determining the age of a user based on audiovisualdata.

Feature data may include edges such as the pixels in a face frame thatdepict the boundaries of a training subject's face or the boundaries ofobjects within the training subject's face, such as the boundaries ofthe training subject's eyes, nose, ears, mouth, cheeks, eyebrows, etc.Feature data may also include the portions of the face frame that depictdifferent objects within the training subject's face, such as thetraining subject's eyes, nose, ears, mouth, cheeks, eyebrows, etc.Objects may be identified using edge detection or by identifying stableregions using a scale-invariant feature transform (SIFT), speeded uprobust features (SURF), fast retina keypoint (FREAK), binary robustinvariant scalable keypoints (BRISK), or any other suitable computervision techniques. The feature data may include feature vectors thatdescribe attributes of an object or edge, such as RGB pixel values forthe object, the position of the object within the face frame, the sizeof the object relative to the face frame, the shape of the object, thetype of object (e.g., nose, eyes, ears, mouth, etc.), or any othersuitable attributes. Feature data may also include pixel distancesbetween particular objects such as a mouth and nose.

Movement data may include the difference in the positions of featuresover multiple frames and/or the rate of change in the positions of thefeatures. For example, video may be captured with a particular framerate (e.g., 24 frames per second). If a particular features moves anaverage of 20 pixels over 48 frames the movement data may indicate thatthe features moved at a rate of 10 pixels per second.

FIG. 4 depicts an exemplary image analysis 400 of feature data overseveral video frames. In some embodiments, the image analysis 400 may beperformed by the biometric characteristic server 102. For example, thevideo frames may be face frames 402-410 of training subject John Doe andthe biometric characteristic server 102 may perform an image analysis ofeach face frame 402-410 to identify feature data and movement data overthe set of face frames. In the first face frame 402, the biometriccharacteristic server 102 extracts several features including thetraining subject's eyes 402 a, ears 402 b, nose 402 c, mouth 402 d, andeyebrows 402 e. The features may be stored as feature vectors thatinclude attributes of each feature, such as RGB pixel values andcorresponding positions of each pixel, the position of the featurewithin the face frame, the size of the feature relative to the faceframe, the shape of the feature, the type of feature, etc. The featuredata may also include distances between features, such as a distancebetween the training subject's eyes 402 a or a distance between thetraining subject's eyes 402 a and eyebrows 402 e. Furthermore, thefeature data may include other geometric properties such as ratios ofsizes of features, lengths of features, widths of features,circumferences of features, diameters of features, or any other suitableproperties.

In the image analysis 400, movement data may also be identified based ona change in position, orientation and/or size of one or several featuresover multiple face frames. For example, in the second face frame 404, afeature 404 c may be identified having similar properties to the feature402 c in the first face frame 402, which may depict the trainingsubject's nose. However, the feature 404 c in the second face frame 404c may be higher than the feature 402 c in the first face frame 402.Therefore, the movement data may indicate that the training subject'snose moved upward by a particular amount of pixels per frame or persecond indicating that the training subject scrunched his nose. In someembodiments, movement data may be relative to changes in position,orientation and/or size of the other features. For example, in the thirdface frame 406, the training subject appears to tilt his head such thateach of the features moves by the same or similar amount. From thisframe, the movement data may indicate that the entire face moved, butthe facial features did not move relative to each other.

The fourth and fifth face frames 408, 410 illustrate additional examplemovements which may be included in the movement data. More specifically,in the fourth frame 408 the training subject's right eye 408 a issmaller than in the first frame 402 indicating the training subject maybe squinting or winking. When both eyes decrease in size at the sametime by a similar amount, the biometric characteristic server 102 maydetermine the training subject is blinking. In the fifth frame 410 thetraining subject's eyebrows 410 e are higher than in the first frame 402which may indicate that the training subject raised his eyebrows 410 e.

For a particular biometric characteristic (e.g., gender), the trainingmodule 134 may classify the feature data and movement data into one ofseveral subsets of training data, where each subset corresponds to adifferent biometric characteristic value (e.g., female) or range ofbiometric characteristic values (e.g., ages 20-29). Once each featurevector, movement vector, etc., is classified into one of the subsets,the training module 134 may analyze each of the subsets to generate astatistical model for determining the particular biometriccharacteristic. For example, when the machine learning technique isneural networks or deep learning, the training module 134 may generate agraph having input nodes, intermediate or “hidden” nodes, edges, andoutput nodes. The nodes may represent a test or function performed onfeature data or movement data and the edges may represent connectionsbetween nodes. In some embodiments, the output nodes may includeindications of biometric characteristic values for a particularbiometric characteristic, such as a different age at each output node orlikelihoods of biometric characteristic values, such as a likelihood ofa particular age. In some embodiments, the edges may be weightedaccording to a strength of the test or function for the preceding nodein determining the biometric characteristic.

For example, a neural network may include four input nodes representingdifferent types of feature data and/or movement data, such as the shapeof a feature, the size of the feature, an initial position of thefeature within the face frame, and a change in position of the featureover a threshold number of frames. The input nodes may be connected toseveral hidden nodes that are connected to an output node indicating theuser is 38 years old. The connections may have assigned weights and thehidden nodes may include tests or functions performed on the featuredata and/or movement data. In some embodiments, the hidden nodes may beconnected to several output nodes each indicating a different biometriccharacteristic value (e.g., a different age). However, this is merelyone example of the inputs and resulting output of the statistical modelfor determining a biometric characteristic. In other examples, anynumber of input nodes may include several types of feature data andmovement data. Additionally, any number of output nodes may providedifferent biometric characteristic values.

Moreover, while biometric characteristics such as age and gender may bepredicted using feature data from a single face frame, more complexbiometric characteristics such as BMI, GSR, blood pressure, bodytemperature, and heart rate may require movement data to generateaccurate models. For example, heart rate may be determined based onsubtle, periodic head motions due to the cyclical movement of blood fromthe heart to the head via the abdominal aorta and carotid arteries.Therefore, periodic frame-to-frame movements of a particular featurewithin the face frames, such as an object or edge that are within thefrequency range of a typical heart rate (e.g., between 0.5 and 5 Hz) maybe indicative of the user's heart rate. In this example, the periodicframe-to-frame movements of a particular feature may be included ininput nodes of a neural network for determining heart rate and may betested at intermediate or “hidden” nodes of the neural network. However,this is merely one example type of movement data that may be used togenerate a statistical model for determining a particular biometriccharacteristic. Additional or alternative types of movement data may beused in combination with feature data from the training subjects togenerate the statistical models for each biometric characteristic. Insome embodiments, portions of face frames may be amplified or magnifiedto identify small frame-to-frame movements (e.g., a movement of aparticular feature of one pixel or a half of a pixel).

As additional training data is collected, the weights, nodes, and/orconnections may be adjusted. In this manner, the statistical models areconstantly or periodically updated to reflect at least a near real-timerepresentation of the feature data and/or movement data.

In addition to generating the statistical model based on feature dataand movement data from face frames, the statistical model may also begenerated based on biometric characteristics determined from otherstatistical models. For example, the statistical model for determiningBMI may be based on feature data and movement data from face frames aswell as age as determined by the statistical model for determining age.Moreover, the statistical model may be generated based on voice datafrom the training subjects. Some of the videos may include an audiocomponent that includes a training subject's voice. Voice data may beindicative of certain biometric characteristics, such as smoking status.The voice data may include several voice components extracted from atraining subject's speech such as frequency, pitch, intensity, tone,etc., which may be used as acoustic vectors in the statistical model.For example, a frequency analysis of the voice data may be performed(e.g., using a Fast Fourier Transformation (FFT) or other frequencytransform) to identify the voice components. In this example, thestatistical model for determining smoking status may be based on featuredata and movement data from face frames as well as voice data includingacoustic vectors. More specifically, the input nodes to the neuralnetwork for determining smoking status may include feature vectors forimage features included in face frames, movement vectors indicative ofthe rate of change of the position or size of image features included inthe face frames, and acoustic vectors indicative of the user's voice.

A statistical model may also be generated for determining emotionalstate. For example, the user's GSR may be indicative of stress levels ofthe user and thereby the user's emotional state. The user's GSR may alsobe combined with the user's heart rate and/or facial expressionsidentified in the feature and movement data of the face frames toidentify the user's emotional state. In some embodiments, thestatistical model for determining emotional state may be generated basedon any suitable combination of GSR, heart rate, feature data, movementdata, voice data, or any other biometric characteristic. As in thestatistical models for determining biometric characteristics, thestatistical model for determining emotional state may be trained usingaudiovisual data representing faces of training subjects havingbiometric characteristics known to the system and based on knownemotional states of the training subjects. For example, the biometriccharacteristic server 102 may receive an indication that a firsttraining subject suffers from clinical depression, a second trainingsubject suffers from anxiety disorder, and a third training subject doesnot have any psychological disorders.

In any event, a statistical model may be generated for each biometriccharacteristic including age, gender, BMI, blood pressure, heart rate,GSR, smoking status, body temperature, etc. In addition to generatingstatistical models for determining biometric characteristics, thebiometric characteristic server 102 may store lookup tables or a set ofrules that correlate biometric characteristic values or a range ofbiometric characteristic values for a particular biometriccharacteristic with an individual health indicator or score. The user'semotional state may also be correlated with an individual healthindicator or score. For example, people between the ages of 1-9 may beassigned an individual health score of 1; people between the ages of10-19 may be assigned an individual health score of 2; people betweenthe ages of 20-29 may be assigned an individual health score of 3, etc.The biometric characteristic server 102 may also store a lookup table orset of rules for combining individual heath indicators or scores togenerate an overall health indicator or score indicative of the overallhealth of the user. For example, the individual health scores may beaggregated or averaged. In another example, the individual health scoresmay be weighted based on the corresponding biometric characteristic andthen aggregated, multiplied, averaged, etc. More specifically, BMI mayhave less of an effect on overall health than blood pressure and thus,an individual health score assigned to blood pressure may be weightedhigher than an individual health score assigned to BMI.

Furthermore, the biometric characteristic server 102 may store lookuptables or a set of rules that correlate overall health indicators orscores or ranges of overall health scores with a longevity metric, suchas an estimate of the user's remaining life expectancy or longevity. Forexample, people with overall health scores above 90 may be expected tolive 80 more years; people with overall heath scores between 80 and 89may be expected to live 70 more years; people with overall health scoresbetween 70 and 79 may be expected to live 60 more years, etc. Thebiometric characteristic server 102 may also store a set of rules forproviding a monthly or yearly life insurance premium quote based on theuser's remaining life expectancy, the coverage amount, and the policytype (e.g., term life insurance or whole life insurance).

In other embodiments, the individual health indicator or score, overallhealth indicator or score, and/or longevity metric may be determinedusing machine learning techniques based on the biometric characteristicvalues for training subjects and known health or longevity data for thetraining subjects.

In any event, the training module 134 may then test each statisticalmodel generated using neural networks, deep learning, naïve Bayes,support vector machines, linear regression, polynomial regression,logistic regression, random forests, boosting, nearest neighbors, or anyother suitable machine learning technique. For example, the trainingmodule 134 may obtain test data including test video frames depicting atest subject and test biometric characteristics of the test subject.While the test biometric characteristics are known (e.g., the biometriccharacteristics of the test subject are provided to the biometriccharacteristic server 102 but the test video frames and test biometriccharacteristics are used for testing purposes), the training module 134may determine a biometric characteristic value for each biometriccharacteristic for the test subject by extracting feature and movementdata from the test video frames and comparing the feature and movementdata to the respective statistical models.

For example, when a statistical model for determining BMI is a neuralnetwork, the training module 134 may traverse nodes of the neuralnetwork using the feature and movement data from the test video frames.After traversing each of the nodes which correspond to the test featureand movement data, the training module 134 may reach an output nodewhich may indicate a BMI value, such as 21. The BMI value determined bythe training module 134 may then be compared to the test BMI. In someembodiments, if the BMI value determined by the training module 134 iswithin a threshold amount of the test BMI (e.g., ±3), the determinationmay be deemed correct.

In any event, when the training module 134 is correct more than apredetermined threshold amount of time, the statistical model for theparticular biometric characteristic may be provided to the biometricidentification module 136. The statistical models for each biometriccharacteristic may be tested and presented to the biometricidentification module 136. On the other hand, if the training module 134does not correctly determine biometric characteristic values for aparticular biometric characteristic more than the threshold amount, thetraining module 134 may continue obtaining sets of training data fortraining subjects to further train the statistical model correspondingto the particular biometric characteristic.

When each of the statistical models have been provided to the biometricidentification module 136, the biometric identification module 136 mayreceive video or other audiovisual data from a user, where the user'sbiometric characteristics are unknown to the biometric characteristicserver 102. Accordingly, the biometric identification module 136 mayapply feature data and movement data from face frames included in thereceived video to each of the statistical models generated by thetraining module 134 to determine biometric characteristic values for theuser and/or an emotional state of the user. The biometric characteristicvalues and/or emotional state may be used to determine individual healthindicators or scores, which may in turn be used to determine an overallhealth indicator or score and a longevity metric. The longevity metricmay include an estimate of the user's remaining life expectancy and/or amonthly or yearly life insurance premium quote based on the user'sremaining life expectancy, a coverage amount, and a policy type (e.g.,term life insurance or whole life insurance). The biometricidentification module 136 may then provide an indication of thebiometric characteristics to the user's client computing device 106-116for display to the user. The indication may include the biometriccharacteristic values, the individual health scores, the overall healthscore, the longevity metric, or any suitable combination thereof. Thisis described in more detail below.

FIG. 5 depicts an exemplary video capturing screen 500 which may begenerated by the biometric characteristic server 102 and displayed bythe client application 266 of the client computing device 112. In otherembodiments, the exemplary screen may be generated and displayed by theclient computing device 112. As will be appreciated by those of ordinaryskill in the relevant art(s), the exemplary video capturing screen shownin FIG. 5 is for illustrative purposes, and the associated functionalitymay be implemented using any suitable format and/or design forfacilitating corresponding described functionalities without departingfrom the spirit and scope of the present disclosure. In someembodiments, the biometric characteristic server 102 may transmit webpages.

The client application 266 may include a home screen (not shown) thatprompts the user to capture video of herself to receive an indication ofher biometric characteristics, health status, and/or longevity withoutfilling out a form or providing textual information regarding herhealth. For example, the client application 266 may be associated withproviding life insurance and the captured video may be provided insteadof filling out a life insurance application. To receive a monthly oryearly life insurance premium quote, the user may simply capture a shortvideo of herself (e.g., for five seconds, ten seconds, a minute, etc.)and provide the video to the biometric characteristic server 102 via auser control, such as a “Submit” button. The user may be requested toprovide some information regarding the life insurance such as the policytype (e.g., term life insurance or whole life insurance) and thecoverage amount (e.g., $500,000). The user may also be requested tospeak during the video, so that the biometric characteristic server 102may provide an audio analysis of the user's voice. In some embodiments,the user is prompted to state the policy type and coverage amount beingrequested during the video.

The home screen (not shown) may include user controls for capturing anew video or retrieving a video previously stored at the clientcomputing device 112. The video capturing screen 500 may be presented inresponse to the user selecting the user control for capturing a newvideo. The video capturing screen 500 may include a record button 502for starting and stopping the recording, a timer 504 indicating thelength of time of the recording, and a camera view 506 presenting thevideo as it is being recorded. In some embodiments, the user may selectthe record button 502 once to start the recording and a second time tostop the recording. The client application 266 may provide instructionsfor the user to record for a particular amount of time (e.g., tenseconds), within a particular time range (e.g., between five and tenseconds), or for at least a threshold amount of time (e.g., fiveseconds). The user may stop the recording after a sufficient amount oftime, as indicated by the timer 504. If the recording lasts for theparticular amount of time, time range, or for at least the thresholdamount of time, the client application 266 may present a user control(not shown) for providing the video to the biometric characteristicserver 102, such as a “Submit” button. In some scenarios, the user maydelete the recording and record a new video.

The biometric characteristic server 102 and more specifically, thebiometric identification module 136 may receive the video and identifyface frames within each video frame using face detection techniques asdescribed above. Then the biometric identification module 136 mayextract feature data and movement data from the face frames depictingthe user and may apply the feature data and movement data to each of thestatistical models for determining biometric characteristics and/or anemotional state of the user.

For example, when a statistical model for determining BMI is a neuralnetwork, the biometric identification module 136 may traverse nodes ofthe neural network using the feature and movement data from the faceframes depicting the user. After traversing each of the nodes whichcorrespond to the feature and movement data, the biometricidentification module 136 may reach an output node which may indicate aBMI value, such as 32.

In some embodiments, the biometric identification module 136 maygenerate a user profile for the user and may store each of thedetermined biometric characteristics and/or emotional state for the userin the user profile. Also in some embodiments, the user may provide hername (e.g., in the video) and the name may also be stored in the userprofile. In this manner, if the same user submits a subsequent video,the biometric characteristic server 102 may retrieve at least some ofthe user's biometric characteristics from her user profile. An image ofthe user such as a driver's license photograph may also be stored in theuser profile and the image may be compared with the video frames toverify that the user is the person she claims to be. Additionally, thebiometric characteristics stored in the user profile may be used asadditional training data if it is determined that the biometriccharacteristics are accurate.

In any event, in response to providing the video to the biometriccharacteristic server 102 via the user control, the client application266 may display a health status screen (not shown) which may include anindication of the biometric characteristics of the user as determined bythe biometric characteristic server 102 via the statistical models. Theindication of the biometric characteristics may include biometriccharacteristic values for the user. The indication of the biometriccharacteristics may also include an indication of the user's emotionalstate. For example, the health status screen may indicate the user is a26 year-old female with a BMI of 19, a heart rate of 60 beats perminute, a blood pressure of 120/80, a body temperature of 98.2° F., etc.Furthermore, the indication of the biometric characteristics may includeindividual health scores for each of the biometric characteristics, anoverall health score for the user, and a longevity metric such as themonthly or yearly life insurance premium quote for a particular policytype and coverage amount. The user may then purchase the life insurancevia a user control such as an “Accept” button associated with themonthly or yearly life insurance premium quote.

In some embodiments, the user may review the biometric characteristicvalues and confirm that the biometric characteristic values are accuratevia a user control or may edit the biometric characteristic values. Forexample, the health status screen may indicate the user is 31 years old.If she is in fact 29 years old, she may adjust her age via a text fieldor drop-down menu. Then the adjusted age may be provided to thebiometric characteristic server 102 to recalculate her overall healthscore and longevity metric. If the biometric characteristic values areaccurate then the overall health score and longevity metric may not needto be adjusted.

Additionally, when the user's overall health score is below a thresholdamount or a particular biometric characteristic value is outside athreshold range, the client application 266 may display a warning ornotification to the user and/or may advise the user to see a healthcareprofessional. In some embodiments, the user's emotional state may factorinto the user's overall health indicator or longevity metric. Forexample, users who are depressed may have lower overall health scores.If the user is identified as depressed, suffering from anotherpsychological disorder, or at risk for suicide according to thedetermined emotional state, the warning or notification may indicate tothe user that she may be suffering from a psychological disorder andshould seek help immediately. In some embodiments, upon receivingpermission from the user, the biometric characteristic server 102 maysend the warning or notification to emergency personnel or a healthcareprofessional identified by the user.

In other embodiments, the user does not provide the policy type andcoverage amount and several life insurance premium quotes are providedfor several different combinations of policy types and coverage amounts.For example, the client application 266 may display a table (not shown)where one column is for term life insurance, another column is for wholelife insurance, each row is for a different coverage amount, and theentry corresponding to a particular row and column (e.g., whole lifeinsurance for $100,000 of coverage) includes the monthly or yearly lifeinsurance premium quote for the policy type and coverage amount.

In an exemplary scenario, Jane Smith would like to receive a lifeinsurance premium quote for a whole life policy and a coverage amount of$1,000,000. Without providing any textual information regarding herhealth status or biographical information, Jane takes a ten second videoof herself with her smart phone while explaining the type of lifeinsurance policy she would like and the coverage amount in the video.The video is then provided to the biometric characteristic server andfeature data and movement data extracted from face frames within thevideo are compared to statistical models for determining biometriccharacteristics using one or more of the machine learning techniquesdescribed above. The biometric characteristics are then applied to datafrom lookup tables or additional statistical models for determining alongevity metric for Jane Smith. Jane's smart phone may then receive anddisplay a monthly or yearly life insurance premium quote for a wholelife policy having a coverage amount of $1,000,000 based on herdetermined biometric characteristics. In this manner, Jane may receive alife insurance premium quote in real-time or at least near real-timeupon providing the video and the biometric characteristic server mayaccurately determine her biometric characteristics, thereby reducing therisk of obtaining false information.

Additionally, if any of Jane's biometric characteristics are outside ofa healthy range (e.g., a predetermined range of acceptable values) ormore than a threshold variance from the healthy range, she may receive anotification indicating the biometric characteristic that is outside ofthe healthy range, the corresponding biometric characteristic value,and/or an indication of the healthy range. The notification may alsoadvise her to see a healthcare professional. If Jane's emotional stateindicates she is suffering from a psychological disorder, she mayreceive a notification indicating the psychological disorder and arecommendation to see a healthcare professional.

FIG. 6 depicts a flow diagram representing an exemplary method 600 foridentifying biometric characteristics of a user based on audiovisualdata. The method 600 may be executed on the biometric characteristicserver 102. In some embodiments, the method 600 may be implemented in aset of instructions stored on a non-transitory computer-readable memoryand executable on one or more processors of the biometric characteristicserver 102. For example, the method 600 may be performed by the trainingmodule 134 and the biometric identification module 136 of FIG. 1 .

At block 602, the biometric characteristic server 102 and morespecifically, the training module 134 may obtain training data includingsets of audiovisual data of training subjects having known biometriccharacteristics at the time the audiovisual data is captured (e.g., age,gender, BMI, etc.). The training data may include public audiovisualdata such as movies, television, music videos, etc., featuring famousactors or actresses having biometric characteristics which are known orwhich are easily obtainable through public content (e.g., via IMDb®,Wikipedia™, etc.). In some embodiments, the emotional states of thetraining subjects may also be obtained.

A set of audiovisual data for a training subject may be a video of thetraining subject that includes several video frames. In someembodiments, each video frame may be analyzed using face detectiontechniques to identify a portion of each video frame that depicts atraining subject's face (a face frame) (block 604). Face detectiontechniques may include edge detection, pixel entropy, blink detection,motion detection, skin color detection, any combination of these, or anyother computer vision techniques. In some embodiments, the trainingmodule 134 may filter out the remaining portion of the video frame thatdoes not include the training subject's face.

At block 606, each face frame may be further analyzed to identifyfeature data within the face frames. Movement data may also beidentified indicating a change in the positions or sizes of featuresover multiple face frames for the same training subject. Feature datamay include edges such as the pixels in a face frame that depict theboundaries of a training subject's face, the boundaries of objects, orthe objects within the training subject's face. Objects may beidentified using edge detection or by identifying stable regions using ascale-invariant feature transform (SIFT), speeded up robust features(SURF), fast retina keypoint (FREAK), binary robust invariant scalablekeypoints (BRISK), or any other suitable computer vision techniques. Thefeature data may include feature vectors that describe attributes of anobject or edge, such as RGB pixel values for the object, the position ofthe object within the face frame, the size of the object relative to theface frame, the shape of the object, the type of object (e.g., nose,eyes, ears, mouth, etc.), or any other suitable attributes. Feature datamay also include pixel distances between particular objects such as amouth and nose.

Movement data may include the difference in the positions of featuresover multiple frames and/or the rate of change in the positions of thefeatures. For example, video may be captured with a particular framerate (e.g., 24 frames per second). If a particular features moves anaverage of 20 pixels over 48 frames the movement data may indicate thatthe features moved at a rate of 10 pixels per second.

The feature data and the movement data for a training subject may thenbe stored in association with the biometric characteristics of thecorresponding training subject. In some embodiments, voice data may alsobe included within the videos and the training module 134 may extractvoice components from a training subject's speech such as frequency,pitch, intensity, tone, etc.

Then at block 608, a statistical model may be generated for determiningeach of the biometric characteristics (e.g., age, gender, BMI, bloodpressure, heart rate, GSR, smoking status, body temperature, etc.) byanalyzing the training data using various machine learning techniques,such as neural networks, deep learning, naïve Bayes, support vectormachines, linear regression, polynomial regression, logistic regression,random forests, boosting, nearest neighbors, etc. For a particularbiometric characteristic (e.g., gender), the training module 134 mayclassify the feature data, movement data, audio data, or biometriccharacteristic value for another biometric characteristic (e.g., BMI)into one of several subsets of training data, where each subsetcorresponds to a different biometric characteristic value (e.g., female)or range of biometric characteristic values. Once each feature vector,movement vector, acoustic vector, etc., is classified into one of thesubsets, the training module 134 may analyze each of the subsets togenerate a statistical model for determining the particular biometriccharacteristic.

In addition to generating statistical models for determining each of thebiometric characteristics using machine learning techniques, thetraining module 134 may generate a statistical model for determiningemotional state using machine learning techniques. In some embodiments,the statistical model for determining emotional state may be generatedbased on any suitable combination of GSR, heart rate, feature data,movement data, voice data, or any other biometric characteristic.

The training module 134 may also obtain correlations between biometriccharacteristics and/or emotional state and individual health indicatorsor scores from lookup tables, a set of rules, or statistical modelsusing machine learning techniques. Further, the training module 134 mayobtain a set of rules for generating an overall health indicator orscore and a longevity metric based on the individual health indicatorsor scores from a lookup table or statistical models using machinelearning techniques.

At block 610, the biometric identification module 136 may receive a setof audiovisual data for the user, such as video of the user captured fora threshold time period, where the user's biometric characteristics areunknown to the biometric characteristic server 102. For example, thebiometric identification module 136 may receive a video such as thevideo captured by the video capturing screen 500 as shown in FIG. 5 .

The biometric identification module 136 may then identify face frameswithin each video frame using the face detection techniques describedabove and extract feature data and movement data from the face framesdepicting the user. The biometric identification module 136 may alsoobtain voice data from the video. Then the biometric identificationmodule 136 may apply the feature data, movement data, and voice data toone of the statistical models for determining a particular biometriccharacteristic (block 612). This may be repeated for each of thestatistical models to determine each of the biometric characteristicsand the emotional state of the user. In some embodiments, a determinedbiometric characteristic value for one biometric characteristic (e.g.,age) may be applied to a statistical model for determining anotherbiometric characteristic (e.g., BMI).

At block 614, the biometric identification module 136 may determineindividual health indicators or scores from the determined biometriccharacteristics and/or emotional state using the correlations orstatistical model obtained by the training module 134. The individualhealth indicators or scores may be combined or aggregated to generate anoverall health indicator and/or longevity metric using the set of rulesor statistical model obtained by the training module 134 (block 616).

Then the biometric identification module 136 may provide an indicationof the biometric characteristics for display on the user's clientcomputing device (block 618). The indication may include biometriccharacteristic values for the user. The indication of the biometriccharacteristics may also include an indication of the user's emotionalstate. Furthermore, the indication of the biometric characteristics mayinclude individual health scores for each of the biometriccharacteristics, an overall health score for the user, and a longevitymetric such as the monthly or yearly life insurance premium quote for aparticular policy type and coverage amount. For example, the indicationof the biometric characteristics may be presented on a health statusscreen of a client application 266 on the user's client computingdevice, as described above with reference to FIG. 5 . In someembodiments, the biometric identification module 136 may also provide awarning or notification for display on the user's client computingdevice when the user's overall health score is below a threshold amount,a particular biometric characteristic value is outside a thresholdrange, or the user is identified as suffering from a psychologicaldisorder.

FIG. 7 depicts a flow diagram representing an exemplary method 700 forcapturing audiovisual data representing a user and presentingindications of automatically determined biometric characteristics of theuser. The method 700 may be executed on the client computing device 112.In some embodiments, the method 700 may be implemented in a set ofinstructions stored on a non-transitory computer-readable memory andexecutable on one or more processors of the client computing device 112.For example, the method 700 may be performed by the client application266. In other embodiments, the method 700 may be performed by thebiometric characteristic server 102 and/or a combination of the clientcomputing device and the biometric characteristic server 102.

At block 702, the client computing device 112 and more specifically, theclient application 266 may capture video of a user over a threshold timeperiod. For example, the video may be captured via the video capturingscreen 500 as shown in FIG. 5 . In some embodiments, a home screen ofthe client application 266 may include user controls for capturing a newvideo or retrieving a video previously stored at the client computingdevice 112. The client application 266 may also provide instructions forthe user to record for the threshold time period (e.g., five seconds)and/or may provide instructions for the user to speak and answerquestions, such as the user's name, the policy type being requested, andthe amount of coverage being requested. If the recording lasts for thethreshold time period, the client application 266 may present a usercontrol for providing the video to the biometric characteristic server102, such as a “Submit” button.

At block 704, the video is transmitted to the biometric characteristicserver 102, which may in turn analyze the video frames and voice data toextract feature data and movement data within face frames and voicecomponents from the user's speech such as frequency, pitch, intensity,tone, etc. The biometric characteristic server 102 may then applyfeature vectors, movement vectors, and acoustic vectors to statisticalmodels for determining biometric characteristics and an emotional stateof the user. The biometric characteristic server 102 may also determineindividual health indicators or scores, an overall health indicator orscore, and a longevity metric based on the determined biometriccharacteristics and emotional state of the user.

At block 706, the client computing device 112 receives an indication ofthe biometric characteristics of the user from the biometriccharacteristic server 102 without providing textual information to thebiometric characteristic server 102. The indication may includebiometric characteristic values for the user, an indication of theuser's emotional state, individual health scores for each of thebiometric characteristics, an overall health score for the user, and/ora longevity metric such as the monthly or yearly life insurance premiumquote for a particular policy type and coverage amount.

Then the client computing device 112 presents the indication of thebiometric characteristics on the user interface 240. For example, theclient application 266 may include a health status screen which displaysthe indication of the user's biometric characteristics. For example, thehealth status screen may indicate the user is a 26 year-old female witha BMI of 19, a heart rate of 60 beats per minute, a blood pressure of120/80, a body temperature of 98.2° F., etc. The health status screenmay also indicate that for a whole life insurance policy and a coverageamount of $50,000, the yearly life insurance premium will be $700. Theuser may then purchase the life insurance via a user control on thehealth status screen such as an “Accept” button associated with themonthly or yearly life insurance premium quote.

In some embodiments, the user may review the biometric characteristicvalues and confirm that the biometric characteristic values are accuratevia a user control or may edit the biometric characteristic values. Forexample, the health status screen may indicate the user is 31 years old.If she is in fact 29 years old, she may adjust her age via a text fieldor drop-down menu. Then the adjusted age may be provided to thebiometric characteristic server 102 to recalculate her overall healthscore and longevity metric. If the biometric characteristic values areaccurate then the overall health score and longevity metric may not needto be adjusted.

Additionally, when the user's overall health score is below a thresholdamount, a particular biometric characteristic value is outside athreshold range, or the user's emotional state indicates the user issuffering from a psychological disorder, the client application 266 maydisplay a warning or notification to the user and/or may advise the userto see a healthcare professional.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (e.g., code embodiedon a machine-readable medium or in a transmission signal) or hardware.In hardware, the routines, etc., are tangible units capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connect the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method may be performed by one or more processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription, and the claims that follow, should be read to include oneor at least one and the singular also includes the plural unless it isobvious that it is meant otherwise.

This detailed description is to be construed as exemplary only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application.

We claim:
 1. A method for determining biometric characteristics of auser, the method comprising: receiving, by one or more processors andfrom a computing device, sensor data associated with the user;determining, by the one or more processors and based on the sensor data,one or more features associated with the user; passing the one or morefeatures to a machine-trained model; receiving, from the machine-trainedmodel and based on the one or more features, one or more biometriccharacteristics of the user; determining, by the one or more processorsand based on the sensor data, movement data associated with at least onefeature of the one or more features; determining, by the one or moreprocessors and based on the one or more biometric characteristics andthe movement data, a health indicator of the user; determining, by theone or more processors and based on the health indicator, an insurancequote; and transmitting, by the one or more processors and to thecomputing device, the one or more biometric characteristics, the healthindicator, and the insurance quote.
 2. The method of claim 1, furthercomprising: determining, by the one or more processors, an additionalhealth indicator of the user corresponding to the one or more biometriccharacteristics; determining, based at least in part on the healthindicator and the additional health indicator, an overall healthindicator; and transmitting, by the one or more processors and to thecomputing device, the overall health indicator.
 3. The method of claim2, further comprising: determining, by the one or more processors andbased on the overall health indicator, a life expectancy of the user;and determining, by the one or more processors and based on the lifeexpectancy of the user, the insurance quote.
 4. The method of claim 3,further comprising: receiving, by the one or more processors and fromthe computing device, information about an adjustment of at least one ofthe one or more biometric characteristics; determining, by the one ormore processors and based at least in part on the information, anupdated insurance quote; and transmitting, by the one or more processorsand to the computing device, the updated insurance quote to thecomputing device.
 5. The method of claim 1, wherein the sensor datacomprises audio data, the method further comprising: identifying, by theone or more processors and from the audio data, at least one acousticvector as the one or more features, the at least one acoustic vectorbeing associated with at least one of frequency, pitch, intensity, ortone.
 6. The method of claim 5, further comprising: determining at leastone of smoking status or emotional state of the user based on the atleast one acoustic vector.
 7. The method of claim 1, wherein the one ormore biometric characteristics include at least one of: age, gender,body mass index, heart rate, body temperature, galvanic skin response,smoking status, or emotional state.
 8. A system for determiningbiometric characteristics of a user comprising: one or more processors;and a non-transitory computer-readable memory storing thereoninstructions that, when executed by the one or more processors, causethe system to: generate a machine-trained model for determining one ormore biometric characteristics of a user; receive sensor data of theuser from a computing device, the sensor data being captured over a timeperiod; determine, based on the sensor data, one or more featuresassociated with the user; pass the one or more features to themachine-trained model; receive, from the machine-trained model, the oneor more biometric characteristics of the user; determine, based on thesensor data, movement data associated with the one or more features;determine, based on the one or more features and the movement data, ahealth indicator of the user; determine, based on the health indicator,an insurance quote; and transmit the one or more biometriccharacteristics, the health indicator, and the insurance quote to thecomputing device.
 9. The system of claim 8, wherein the instructionsfurther cause the system to: determine an additional health indicator ofthe user corresponding to the one or more biometric characteristics;determine, based at least in part on the health indicator and theadditional health indicator, an overall health indicator; and transmitthe overall health indicator to the computing device.
 10. The system ofclaim 8, wherein to generate the machine-trained model for determiningone or more biometric characteristics of a user, the system is furthercaused to: obtain a set of training data corresponding to a plurality ofpeople, the set of training data including one or more biometriccharacteristics pre-obtained for individual of the plurality of people;and train the machine-trained model using the pre-obtained one or morebiometric characteristics for the individual of the plurality of people.11. The system of claim 8, wherein the sensor data comprises audio dataand to determine, based on the sensor data, one or more featuresassociated with the user, the system is further caused to: identify,from the audio data, at least one acoustic vector as the one or morefeatures associated with at least one of: frequency, pitch, intensity,or tone.
 12. The system of claim 8, wherein to determine, based on thesensor data, one or more features associated with the user, the systemis further caused to: determine at least one of smoking status oremotional state of the user based on the at least one acoustic vector.13. The system of claim 8, wherein the one or more biometriccharacteristics include at least one of: age, gender, body mass index,heart rate, body temperature, galvanic skin response, smoking status, oremotional state.
 14. The system of claim 8, wherein the one or morefeatures associated with the user include at least one of eyes, ears,nose, mouth, or eyebrows, and the movement data associated with the oneor more features indicate changes in at least one of position,orientation, or size of at least one of the one or more features.
 15. Anon-transitory computer-readable memory storing thereon instructionsthat, when executed by one or more processors, cause the one or moreprocessors to: receive a request to quote for an insurance from acomputing device, the request including sensor data of a user; determineone or more features associated with the user based on the sensor data;pass the one or more features to a machine-trained model; receive, fromthe machine-trained model, one or more biometric characteristics of theuser; determine, based on the sensor data, movement data associated withat least one feature of the one or more features; determine, based onthe one or more biometric characteristics and the movement data, ahealth indicator of the user; determine an estimated quote of theinsurance, the estimated quote being determined based on the one or morebiometric characteristics and the health indicator of the user; andtransmitting, by the one or more processors, the estimated quote of theinsurance to the computing device in response to the request.
 16. Thenon-transitory computer-readable memory of claim 15, wherein theinstructions further cause the one or more processors to: determine anadditional health indicator of the user corresponding to the one or morebiometric characteristics; determine, based at least in part on thehealth indicator and the additional health indicator, an overall healthindicator; and transmit, to the computing device, the overall healthindicator to the computing device.
 17. The non-transitorycomputer-readable memory of claim 15, wherein the instructions furthercause the one or more processors to: generate the machine-trained modelfor determining the one or more biometric characteristics of a userincluding: obtaining a set of training data corresponding to a pluralityof people, the set of training data including one or more biometriccharacteristics pre-obtained for individual of the plurality of people;and training the machine-trained model using the pre-obtained one ormore biometric characteristics for the individual of the plurality ofpeople.
 18. The non-transitory computer-readable memory of claim 15,wherein the sensor data comprises audio data, and to determine one ormore features associated with the user based on the sensor data, the oneor more processors are further caused to: identify, from the audio data,at least one acoustic vector as the one or more features associated withat least one of: frequency, pitch, intensity, or tone.
 19. Thenon-transitory computer-readable memory of claim 18, wherein the one ormore processors are further caused to: determine at least one of smokingstatus or emotional state of the user based on the at least one acousticvector.
 20. The non-transitory computer-readable memory of claim 15,wherein the one or more features associated with the user include atleast one of eyes, ears, nose, mouth, or eyebrows, and the movement dataassociated with the one or more features indicate changes in at leastone of position, orientation, or size of at least one of the one or morefeatures.